Adaptive private network with path maximum transmission unit (MTU) discovery process

ABSTRACT

Systems and techniques are described for a path maximum transmission unit (MTU) discovery method that allows the sender of IP packets to discover the MTU of packets that it is sending over a conduit to a given destination. The MTU is the largest packet that can be sent through the network along a path without requiring fragmentation. The path MTU discovery method actively probes each sending path of each conduit with fragmentation enabled to determine a current MTU and accordingly increase or decrease the conduit MTU. The path MTU discovery process is resilient to errors and supports retransmission if packets are lost in the discovery process. The path MTU discovery process is dynamically adjusted at a periodic rate to adjust to varying network conditions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation under 35 U.S.C. § 120 of U.S. patentapplication Ser. No. 15/385,984 filed on Dec. 21, 2016, and to be issuedon Aug. 14, 2018 as U.S. Pat. No. 10,050,898 which is a continuationunder 35 U.S.C. § 120 of U.S. patent application Ser. No. 14/019,723entitled “An Adaptive Private Network with Path Maximum TransmissionUnit (MTU) Discovery Process” filed Sep. 6, 2013 and issued as U.S. Pat.No. 9,584,407 on Feb. 28, 2017, the disclosures of which are herebyincorporated by reference in their respective entireties.

U.S. Pat. No. 8,125,907 filed on Jun. 11, 2009 entitled “Flow-BasedAdaptive Private Network with Multiple WAN-Paths, U.S. Pat. No.8,452,846 filed on Aug. 12, 2011 entitled “Adaptive Private NetworkAsynchronous Distributed Shared Memory Services”, and U.S. Pat. No.9,069,727 filed on Dec. 19, 2012 entitled “An Adaptive Private Networkwith Geographically Diverse Network Control Nodes” have the sameassignee as the present application, are related applications, and arehereby incorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to improved networkcommunication. More specifically, the present invention relates toimproved path maximum transmission unit (MTU) discovery systems andprocesses.

BACKGROUND OF THE INVENTION

The introduction of frame relay in the early 1990's brought lower cost,higher bandwidth, improved reliability, and simpler management controlto enterprise wide area networks (WANs) as compared to X.25 andpoint-to-point leased-line alternatives. Frame relay, together withsingle-source asynchronous transfer mode (ATM) and multiprotocol labelswitching (MPLS) services, still dominate the enterprise WAN market forcorporate Internet traffic. Such traditional standards tend to havefixed MTU sizes which may be changed by a network administrator. Acustomer installs one of these networks and pays a single carrier a feeassociated with the reliability and bandwidth the particular networkprovides. For example, a network may be advertised to provide “3 and ½nines” (99.95%) or better reliability and have a fee based on thisreliability and a cost per megabytes per second (Mbps). The present costfor such a network is almost as high as the fee paid back in 1998.

Wi-Fi is a name for wireless local area network (WLAN) based on the IEEE802.11 set of standards and WiMax is another wireless network based onthe IEEE 802.16 set of standards. WiMax supports a much larger range andhigher data rates as compared to Wi-Fi. With Wi-Fi and WiMax, the MTUsize changes dynamically based on the line of sight between basestations and receivers.

Path MTU discovery allows a sender of Internet Protocol (IP) packets todiscover a maximum transmission unit (MTU) of packets that it may sendto a given destination. According to RFC4821 Packetization Layer PathMTU Discovery document, the maximum transmission unit (MTU) is the sizein bytes of the largest IP packet, including the IP header and payload,that can be transmitted on a link or a path. A link is a communicationfacility or medium over which nodes can communicate at the link layer,i.e., the layer immediately below IP which is either IPv4 or IPv6. Apath through the network is a set of links traversed by a packet betweena source node and a destination node.

If a router tries to forward a packet to an interface whose MTU issmaller than the packet size, the router has two options. The router canfragment the packet into pieces small enough to fit within the MTU or itcan drop the packet. If a don't fragment (DF) bit of the IP header isset, then the router should drop the packet rather than fragment it.Standard RFC 792 defines an Internet control message protocol (ICMP)message of type 3 (destination unreachable), code 4 (fragmentationneeded and DF bit set) that can be returned to the sender of the packetto alert the host that the packet was too large to be transmittedwithout fragmentation.

SUMMARY OF THE INVENTION

Among its several aspects, the present invention recognizes the currentmethod has a number of problems and provides approaches for addressingissues such as those noted below. For example, many network securitydevices may block all ICMP messages for security benefits. As aconsequence, packets having a size greater than the current MTU for thepath may be dropped without providing an indication of the packet loss.In another example, if the network changes and the MTU size increases,network node may not know about the MTU size increase and keeps using aprevious small MTU size, resulting in sub-optimal performance.

The present invention recognizes that it is advantageous to have anaccurate and timely MTU discovery method which will actively probe eachsending path of each conduit to find out the path's current MTU andadjust accordingly. The techniques for addressing such advantages arediscussed further below.

Also, among its several aspects, the present invention addresses systemsand techniques which improve performance, reliability, andpredictability of networks without requiring costly hardware upgrades orreplacement of existing network equipment. To such ends, an embodimentof the invention addresses a method in a network node to dynamicallyadjust a maximum transmission unit (MTU). A path MTU probe packet istransmitted with padding to meet a packet length according to a selectedMTU and allowing the path MTU probe packet to be fragmented if thepacket is too large. A path MTU probe trailer packet is transmittedhaving a packet length significantly smaller than the selected MTU. Apath MTU received probe packet is then determined to be received. Theselected MTU is adjusted up upon determining an Internet protocol (IP)datagram length of the received path MTU received probe packet is thesame as the selected MTU defined in the path MTU probe packet, wherein asubsequent MTU discovery probe utilizes the adjusted MTU.

Another embodiment addresses a method in a network node to dynamicallyadjust a maximum transmission unit (MTU). A path MTU probe packet istransmitted with padding to meet a packet length specified by a selectedMTU and allowing the path MTU probe packet to be fragmented if thepacket is too large. A path MTU probe trailer packet is transmittedhaving a packet length significantly smaller than the selected MTU. Uponreceiving a path MTU received probe packet having an Internet protocol(IP) datagram length the same as the selected MTU defined in the pathMTU probe packet, the selected MTU is adjusted up for a subsequent MTUdiscovery probe of the adjusted MTU.

Another embodiment addresses a method in a network node to dynamicallyadjust a maximum transmission unit (MTU). A path MTU probe packet istransmitted with padding to meet a packet length specified by a selectedMTU and allowing the path MTU probe packet to be fragmented if thepacket is too large. A path MTU probe trailer packet is transmittedhaving a packet length significantly smaller than the selected MTU. Uponreceiving a reply timeout indicating a response to the path MTU probepackets has not been received and with a retry count that is less thanN, the path MTU probe packet is retransmitted with the selected MTU, thepath MTU probe trailer packet is retransmitted, and the retry count isupdated to indicate an additional retransmission has been attempted

Another embodiment addresses a computer readable non-transitory mediumstoring a computer program which causes a computer system to perform amethod in a network node to dynamically adjust a maximum transmissionunit (MTU). A path MTU probe packet is transmitted with padding to meeta packet length specified by a selected MTU and allowing the path MTUprobe packet to be fragmented if the packet is too large. A path MTUprobe trailer packet is transmitted having a packet length significantlysmaller than the selected MTU. Upon receiving a path MTU received probepacket having an Internet protocol (IP) datagram length the same as theselected MTU, the selected MTU is adjusted up for a subsequent probe ofthe adjusted MTU.

A more complete understanding of the present invention, as well as otherfeatures and advantages of the invention, will be apparent from thefollowing detailed description, the accompanying drawings, and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention will become more fully apparentfrom the following description and appended claims, taken in conjunctionwith the accompanying drawings. Understanding that these drawings depictonly exemplary embodiments and are, therefore, not to be consideredlimiting of the invention's scope, the exemplary embodiments of theinvention will be described with additional specificity and detailthrough use of the accompanying drawings in which:

FIG. 1 illustrates an adaptive private network (APN) with APN networkservice paths in accordance with the present invention;

FIG. 2 illustrates an APN conduit service between a control node and aclient node in accordance with the present invention;

FIG. 3 illustrates an APN having an APN network control node (NCN)coupled through sixteen APN conduits to sixteen APN client nodesaccording to the present invention;

FIG. 4A illustrates a successful MTU probe flow in accordance with thepresent invention;

FIG. 4B illustrates for WAN egress, a reception flow for a path MTUprobe trailer only packet in accordance with the present invention;

FIG. 4C illustrates conduit operations when probe packets are lost inaccordance with the present invention;

FIG. 4D illustrates a probe trailer received out of order flow inaccordance with the present invention;

FIG. 5A illustrates an exemplary transport reliable protocol (TRP)control packet formatted for MTU messages in accordance with the presentinvention;

FIG. 5B illustrates an exemplary MTU probe packet in accordance with thepresent invention;

FIG. 5C illustrates an exemplary MTU probe trailer packet in accordancewith the present invention;

FIG. 5D illustrates an exemplary MTU received probe packet in accordancewith the present invention;

FIG. 5E illustrates an exemplary MTU received trailer only packet inaccordance with the present invention;

FIG. 6 is a path MTU discovery state machine in accordance with thepresent invention; and

FIG. 7 illustrates an exemplary MTU search process 700 in accordancewith the present invention.

DETAILED DESCRIPTION

FIG. 1 shows an example of an adaptive private network (APN) 100 inwhich the present invention may be suitably employed as described infurther detail below, including the network components, flows, paths,and services. The APN 100 includes one or more wide area networks(WANs), such as WAN 102, APN appliances 104-106, WAN routers 110 ₁-110₃, and network application services as well as APN conduits between APNappliances, as described in more detail below.

An APN path is a logical connection established between two WAN linkslocated at different geographic sites across a WAN.

An APN conduit is a virtual connection between two APN nodes, formed byaggregating one or more APN paths and their allocated WAN linkresources.

A conduit MTU is a minimum link MTU of the one or more APN paths betweena source node and a destination node.

An APN appliance (APNA) is a device that contains APN node functionalityincluding all software modules within.

A WAN link represents a physical access point to the wide area network(WAN), such as a digital subscriber line (DSL) connection or a cablemodem. The distinctive characteristic of a WAN link is the bandwidth, orin other words, the amount of data capacity available for transmissionand reception. WAN links can be shared among APN conduits, and intranetand Internet network services. In the present embodiments, the APNappliances do not directly attach to WAN links APN appliancescommunicate with WAN links through logical connections, such as the WANrouters 110 ₁-110 ₃ of FIG. 1.

A private WAN link provides a physical access point to non-public WANdestinations. Examples of such private WAN links include an asynchronoustransfer mode (ATM) link with an ATM virtual circuit, a frame relay linkwith a frame relay circuit, a multiprotocol label switching (MPLS)tunnel, a virtual private network (VPN) tunnel, or a leasedpoint-to-point line. Connectivity on a network having a private WAN linkis made to a private list of destinations on the other end of thenetwork. A public WAN link represents a physical access point to theInternet. It can be assumed that any public WAN link can establish aconnection to any other public WAN link.

An APN service is a set of processing steps performed on packets thatare transmitted through the APN. As illustrated in FIG. 1, data trafficthat moves through APN 100 and APN appliance 106 may require differenttypes of services depending on where the sending and receiving stationsare located. An APN service instance is a particular configuredcontextual instance of an APN service held in an APN appliance memory107 internal to the APN appliance 106, for example. An APN serviceinstance's memory contains, but is not limited to, context specificconfiguration data, statistical data, and tracking states data. Forexample, an APN node may have multiple APN conduits that connect toremote APN nodes. For each APN conduit there exists a separate APNservice instance for the APN conduit service type.

An APN conduit service associated with path 112 manages network trafficpackets that are transmitted through the APN 100 from the APN appliance105 through router 110 ₁, through the WAN 102, through another router110 ₃ to APN appliance 104. The APN conduit service for path 112operates on both APN appliances 104 and 105. The APN conduit servicesends and receives data between a first geographic location that has anAPN appliance 105 and a different geographic location that has an APNappliance 104 utilizing the full benefits provided by the APN conduitservice for WAN resource allocation and network adaptation. An APNintranet service associated with path 114 is used to manage the sendingand receiving of data between a first geographic location that has theAPN appliance 105 and a different geographic location within anenterprise non-APN site 120 that does not have an APN appliance by wayof a WAN link that is also utilized by other APN services.

In another embodiment, an APN intranet service, such as the oneassociated with path 112, may be used to send and receive data to andfrom a different geographic location that has an APN appliance, but anadministrator selectively configures the APN not to use the APN conduitservice 112 for a particular type or class of traffic. An APN Internetservice associated with path 116 is used to send and receive databetween a first geographic location that has the APN appliance 105 and adifferent geographic location that is external to an enterprise networkby way of a WAN link that is also utilized by other APN services. Forexample, traffic using the APN Internet service may be associated with anetwork user accessing a public Internet web server 122. An APN passthrough service 118 is used to send and receive data between a firstgeographic location that has an APN appliance 105 and a local site 124within the same first geographic location. In another embodiment, an APNpass through service may be used to send and receive data between afirst geographic location that has the APN appliance 105 and differentgeographic location within an enterprise network that does not have anAPN appliance and does not traverse the WAN using any WAN linksassociated with any other APN services.

FIG. 2 illustrates an APN conduit 2-ended service 200 between an APNnode A 202 and an APN node B 204 according to the present invention.Each APN node contains a collection of software modules which govern itsparticipation within an APN. The software modules for the APN node A 202and the APN node B 204 include control plane modules 210 and 230, WANingress processor modules 212 and 234, WAN egress processor modules 214and 232, and node administrative and interface software program modules276 and 278, respectively. As illustrated in FIG. 2, the WAN ingressprocessor modules 212 and 234 includes conduit services 220 and 222, andWAN egress processor modules 214 and 232 includes a duplicate conduitservice 224 and 226. Intranet service, Internet service, and passthrough service are also provided at each APN node. Each APN servicetype, including conduit, intranet, Internet, and pass through servicetypes, implements processes for each type of data traffic that iscommunicated to and from the WAN respectively.

As illustrated in FIG. 2, APN conduit traffic, identified by bold dashedarrow path 206 and 208, flows through two APN nodes 202 and 204 as thetraffic traverses the APN. WAN ingress processing module 234 of APNclient performs the WAN ingress conduit service processing 222 prior totransmitting the traffic 206 via the WAN 211 to the APN node A 202. WANegress processor module 214 of the APN node A 202 performs the WANegress conduit service processing 224 prior to transmitting the traffic206 to the node or nodes located on LAN 240. The binding of the one APNnode's WAN ingress conduit processing 222 to the peer APN node's WANegress conduit service processing 224 constitutes an APN conduit 244 inwhich traffic is actively monitored and managed across multiple WANresources.

The APN is capable of using disparate asymmetric WAN links which vary inbehavior of bandwidth, latency, jitter, packet loss and congestionfrequently over time. For example, the APN can use an asymmetric DSL WANlink that transmits data at 512 kbps upstream to the WAN and 6 mbps fromthe WAN through the public network combined with a private symmetricleased circuit T1 WAN link that transmits data at 1544 kbps upstream anddownstream and a cable broadband connection that transmits data at 312kbps upstream to the WAN and 3 mbps from the WAN to a peer havingadequate aggregation bandwidth of these rates for a single TCP filetransfer session at a theoretical transmit rate of 2368 kbps and receiveat 10544 kbps. Practically, under good network behavior the actual ratewould approach 90% of these rates. If the behavior of the connection wasto change, for example the paths to the DSL link were to have dramaticlevels of loss, the APN would, using its high frequency performancefeedback mechanism, adapt the network to avoid or mitigate the issues byusing alternative resources or attempting to recover from the loss.

In a presently preferred embodiment, the APN node's software modules ata site are stored and operate in the same physical APN appliance;however, the modules may also exist in separate physical APN appliancesin alternative embodiments. The methods described in connection with theembodiments disclosed herein may be embodied directly in one or moresoftware modules executed by a processor and memory complex such as arack mounted processing device, a personal computer, a server, or thelike having one or more central processing unit devices. The processorand memory complex, for example, may be configured to executeinstructions under control of a software module program stored on acomputer readable non-transitory storage medium either directlyassociated locally with the processor and memory complex, such as may beavailable through an instruction cache, or accessible through an I/Odevice. A software module may reside in a computer readablenon-transitory storage medium which may include random access memory(RAM) memory, flash memory, ROM memory, dynamic random access memory(DRAM), synchronous dynamic random access memory (SDRAM), read onlymemory (ROM), programmable read only memory (PROM), erasableprogrammable read only memory (EPROM), electrically erasableprogrammable read only memory (EEPROM), hard disk, a removable disk, aCD-ROM, digital video disk (DVD), other types of removable disks, or anyother suitable non-transitory storage medium. A non-transitory storagemedium may also be coupled to the processor and memory complex such thatthe hardware processor can read information from, and write informationto, the storage medium over an intranet or the Internet.

An adaptive private network node (APN node) contains software modulesrequired to participate in an adaptive private network. An APN node mayexist in one or more APN appliances at a location. An APN node containsa collection of software modules which govern its participation withinan APN such as in FIG. 2 control plane modules 210 and 230, WAN ingressprocessor modules 212 and 234, and WAN egress processor modules 214 and232. The control plane module is responsible for controlling andparticipating in the control of the APN node in tandem with other APNnodes in the network.

The WAN ingress processor module 212 may suitably be embodied assoftware and hardware components responsible for processing networktraffic for transmission from a local area network (LAN) to a WAN. TheWAN egress processor module 214 may suitably be embodied as softwareoperating on hardware components, such as a processor and memory complexthat is responsible for processing network traffic for transmission froma WAN to a LAN. WAN ingress and WAN egress processor modules arediscussed in further detail below. The APN node's control plane module210 may suitably be embodied as software operating on hardwarecomponents, such as a processor and memory complex that utilizes the APNnode's WAN ingress processor module 212 and WAN egress processor module214 as the means for transmitting and receiving APN node to APN nodecontrol data across the WAN.

Software packages for an APN are distributed through administrativeinterfaces, such as downloading software using interfaces 276 and 278 tothe APN nodes. After a software update, the APN services on the APNnodes 202 and 204 are then restarted thus bringing the APN software nodeconfiguration into synchronization.

FIG. 3 illustrates an APN 300 having an APN network control node (NCN)302 coupled through conduit 320 and through sixteen APN conduits 321-336to sixteen APN client nodes according to the present invention. Asillustrated in FIG. 3, in a presently preferred embodiment, APN 300 iscentrally configured. A network administrator configures the entire APN300 through an APN configuration file that is processed by the NCN 302.The NCN 302 then distributes the configuration settings to all clientnodes in the APN 300. This method of configuring the APN 300 is intendedto provide benefits to the administrator by providing a single point ofconfiguration to the network. It also assures configuration consistencyand compatibility for all APN nodes in the network simultaneously, withstrict version checking. In a presently preferred embodiment, anintensive configuration audit and validation is done to theconfiguration prior to that configuration being applied to the network.This audit greatly decreases risks of invalid configurations beingplaced on the production network. The central configuration alsoprovides for additional configuration bandwidth optimization for thenetwork, by doing a holistic mapping of the APN resources and theirinitial allocations. Furthermore, the centralized configuration canprovide information and warnings to the administrator as to the behaviorof the configuration that may not be obvious or intended from theconfiguration, before loading the configuration onto a productionnetwork.

In one presently preferred embodiment, APN conduits may exist betweenthe NCN and for example sixteen APN client nodes as shown in FIG. 3, forexample, although there is no systemic limit to the number of potentialAPN client nodes. Each APN conduit may have the unique configurationparameters tailored by an administrator for the particular needs of eachgeographic location associated with a particular APN.

For a definition of APN path states, a description of path processingservices is provided below. Any paths currently in a path quality goodstate are eligible to be chosen first. If multiple paths are in a pathquality good state, then an estimated end to end time is evaluated andcompared for each path. If no path is in path quality good state, then apath with the highest bandwidth path quality bad state is chosen.

FIG. 3 is an exemplary APN 300 with geographically diverse nodes inaccordance with the present invention. The exemplary APN 300 isconfigured with sixteen sites 302-318, which are generally locatedremotely from each other. A site would be defined as remote if thedevices are physically in different locations such as differentbuildings, cities, states, time zones or countries. For example, aprimary NCN site 302 may be located in a company's headquarter locationin a first country, a client, such as client 312, may be located insecond country, and the other client sites 304-311 and 313-319 may be atsome locations intermediate between the two other sites. An APNappliance is a device that contains APN node functionality according tosoftware modules, such as the control plane module 210 and 230, the WANingress processor module 212 and 234, and the WAN egress processormodule 214 and 232, as described in more detail above with reference toFIG. 2. The sixteen sites 304-319 are coupled by conduits 321-336,respectively, and each of the conduits provides a configurable virtualconnection between two connected APN appliances. It is noted that whilesixteen client sites 304-319 are illustrated, an APN may support as manyclient sites as are required for the APN.

Each physical network segment in a network has a maximum packet sizeassociated with it and the network devices attached to that segment knowwhat the maximum packet size is, either by way of a configurationparameter or some physical limitation. When a device needs to forward anIP packet larger than the MTU of that segment, it can send an ICMPpacket back and include the MTU of that segment. The networks operatemost efficiently if they do not have to fragment and reassemble packets.For efficient network operation, the entry points in the network createthe largest packets that can be transported that do not requirefragmentation. MTU discovery is about informing the endpoints in thenetwork what the smallest MTU is between the endpoints so thatfragmentation can be avoided on the path from a source node to adestination node. When an APN node receives an ICMP packet specifying anMTU, the received MTU specifies the largest conduit packet that can betransmitted by the APN node. The received MTU is not used directly butis adjusted to include the transport reliable protocol (TRP) overheadand for blocking needed for encryption. The adjusted MTU that isselected is the length of the largest user packet that can be sentwithout having to fragment the packet. For the purposes of the conduitMTU, the APN node examines all the paths of the conduit because theconduit MTU needs to be the smallest of all the path MTUs on theconduit. The conduit MTU allows retransmission of packets on differentpaths in the conduit without having to fragment the packets.

In one embodiment, a path MTU discovery feature begins a path MTUdiscovery cycle every K minute interval, where K is a selectable valueusually between 1 and 30 minutes, such as setting K equal to tenminutes. A value of K=0 indicates the path MTU discovery feature isturned off. During each discovery cycle, which results in one MTUdiscovery instance, the WAN ingress starts the discovery cycle bysending a configured MTU on each send path associated with the WANingress node using a path MTU probing network protocol. The MTU probingnetwork protocol begins with sending a path MTU probe packet having thesize indicated by the configured MTU followed by a path MTU probetrailer packet that is a short packet, much less than the configured MTUindicates. The configured MTU is set in the configuration process forthe network as described with regard to FIG. 3 above. The systemdefaults to 1500 bytes, for example, which is the maximum MTU for anEthernet segment. Many WANs have additional encapsulation layers, whichmay require the customer to manually configure the MTU to a lower value.Multiple path MTU probes and path MTU probe trailer packets may be sentduring a discovery cycle until a largest MTU that can get through to theother side is discovered.

If a customer want to turn off the path MTU discovery feature, thecustomer would change the path MTU discovery interval K in minutes to 0in an adaptive private network (APN) user variable file and cause avariable load instruction to load the interval K=0 in an interval Ktimer to turn the feature off. One reason for disabling the path MTUdiscovery feature, for example, is that a WAN link with small MTU sizes,such as wireless links or satellite links, may reduce the MTU size forthe conduit to a generally unacceptable minimum size. Some networks setthe MTU size to a low value, such as 512 bytes. Generally, such a valueis too small and it is better to not use that path than to try to usethat path and set the conduit MTU to 512 bytes. Such WAN links areunusual, but there are enough of them in use to warrant having a way toaccount for them, such as disabling the path MTU discovery process.

The path MTU discovery feature uses a network protocol having four typesof packets which are also referred to as messages. Two of the packetsare request packet types and include a path MTU probe request and a pathMTU probe trailer request. The other two types of packets are replymessage types and include a path MTU received probe packet and a pathMTU received trailer only packet. FIGS. 4A-4D illustrate path MTUdiscovery operations using a selection of the four types of packets.

FIG. 4A illustrates a successful MTU probe flow 400 in accordance withthe present invention. FIG. 4A illustrates an APN node A 404, an APNnode B 406, and a WAN conduit 408 connecting the two nodes. The APN nodeA 404 has a WAN ingress processor, such as the WAN ingress processormodule 212, and a WAN egress processor, such as the WAN egress processormodule 214. The APN node B 406 has a WAN ingress processor, such as theWAN ingress processor module 234, and a WAN egress processor, such asthe WAN egress processor module 232. The processing of the path MTUpackets is accomplished within a control plane module, such as thecontrol plane module 210. Paths are unidirectional so they have asender, WAN ingress side, and a receiver, WAN egress side. References to“the path” refers to a path being probed for discovery of the MTU anddoes not refer to the path that response packets are taking through thenetwork. The processing of the path MTU packets occurs in the controlplane on the WAN ingress side of the path and the control plane on theWAN egress side of the path. This is advantageous because, the WANingress processor generally has no way to receive packets such as thepath MTU received probe because it does not receive packets from theWAN.

The path MTU discovery feature begins with the control plane on the WANingress side of the path in APN node A sending a path MTU probe requestpacket 410 to the WAN conduit 408. The path MTU probe request packet 410is a size as specified by the MTU contained in the path MTU proberequest packet 410. After traveling through the conduit, a path MTUprobe request packet 412 leaves the WAN conduit 408 to be received bythe control plane on the WAN egress side of the path in APN node B 406.The control plane on the WAN ingress side of the path in APN node Asends to the WAN conduit 408 a path MTU probe trailer 414 having a smallsize less than the MTU. The path MTU probe trailer 414 would have a sizesmaller than the minimally acceptable MTU. For example, if the MTU isnever to be smaller than 512 bytes, then the MTU probe trailer would beset to be smaller than 512 bytes. The path MTU probe request packet 412is processed in the APN node B 406, for example in the control planemodule 230, and a path MTU received probe packet 416 is sent by thecontrol plane from the APN node B 406 to the WAN conduit 408. Aftertraveling through the conduit, a path MTU received probe packet 418leaves the WAN conduit 408 to be received by the control plane in theAPN node A 404. The receipt of the path MTU received probe packet 418indicates the MTU sent with the path MTU probe 410 is acceptable. Thepath MTU probe trailer 420 that leaves the WAN conduit 408 is receivedin the APN node B 406 and is ignored since the path MTU probe 412 waspreviously received.

Advantageously, the path MTU probe request packet 410 is configured withthe do not fragment bit set to off. Since the do not fragment bit isoff, the path MTU probe request packet 410 may be fragmented ifnecessary. For example, when the control plane on the WAN egress side ofthe path in the APN node B 406 receives the first fragment of afragmented path MTU probe request packet 412, the control plane module230 writes the received MTU size selected from the fragmented path MTUprobe request packet 412 so that it can be compared to the selected MTU,as described in more detail below. The actual IP datagram size of thepath MTU probe request packet 412, whether it is fragmented or not,received by node B is written in the path MTU received probe packet 416being sent back to the APN node A 404. The APN node A 404 upon receiptof the path MTU received probe packet 418 determines, in the controlplane module 210 whether the sent MTU was too large and the fragment IPdatagram size is used as the next MTU size by the forwarding code,thereby shortening the path MTU discovery process.

The IP datagram length is generally equal to the selected MTU whichmeans that what was sent from a source node was received at adestination node. If the IP datagram length is less than the selectedMTU, then the sent probe packet was fragmented. Since the probe packetis fragmented, the IP datagram length is within a threshold range of anoptimum setting for the path.

FIG. 4B illustrates for WAN egress only a reception flow 425 for a pathMTU probe trailer only packet in accordance with the present invention.Most network equipment fragment packets that are too large, but that isnot guaranteed. Some network equipment may drop packets that are toolarge. Such a situation is illustrated by the reception flow 425 of FIG.4B. For example, the MTU probe flow 400 continues with larger MTU valuesand packets of the specified MTU size until the MTU size is too largecompared to the actual MTU. As a consequence of the MTU size being toolarge, the path MTU probe request packet 410 is not received at the APNnode B 406 since it was stopped, shown as a large X 411 in FIG. 4B,someplace within the WAN conduit 408. Even though the path MTU proberequest packet 410 was too large, the path MTU probe trailer 414 issmall enough to pass through the WAN conduit 408. The path MTU probetrailer 420 leaves the WAN conduit 408 and is received in the WAN egressprocessor in the APN node B 406. The path MTU probe trailer 420 isprocessed in the APN node B 406, such as by control plane module 230,and a path MTU received probe trailer only packet 426 is sent by thecontrol plane from the APN node B 406 to the WAN conduit 408. Aftertraveling through the conduit, a path MTU received probe trailer onlypacket 428 leaves the WAN conduit 408 to be received by the controlplane in the APN node A 404. The receipt of the path MTU received probetrailer only packet 428 indicates the MTU sent with the path MTU probe410 is not acceptable. The selected MTU is adjusted down when theprobing side received a path MTU probe trailer only packet which meansthe destination node did not receive the probe with the selected MTUlength.

FIG. 4C illustrates conduit operations 450 when probe packets are lostin accordance with the present invention. If there are network problems,packets may be lost across the network for reasons unrelated to the MTUsize. When the APN node A 404 suspects that a path MTU probe requestpacket and probe trailer packet were lost because of network connectionproblems, the control plane on the WAN ingress side of the path sends apath MTU probe request packet followed by a path MTU probe trailerpacket sequence N more times, such as N set to four. FIG. 4C illustratesa first sequence of probe packets 452 and a second sequence of probepackets 454. With the first sequence of probe packets 452 a replytimeout counter is started which upon a pre-specified time havingelapsed as indicated by the timeout counter, a no reply indication ismarked for future use. The reply timeout counter is initialized for eachprobe cycle, such as setting it to zero, for example. After N replytimeouts, as illustrated scenario of FIG. 4C for N equal to four, allprobe packets are considered lost in the WAN conduit 408. Since no pathMTU received probe packet was returned, the last known good MTU ismaintained. N reply timeouts indicate that there is a different problemthan an MTU problem, because even the small trailers could not make itthrough. From an MTU discovery perspective, no further action is takenbecause no new information has been discovered. From a networkperspective, the control plane would be detecting heavy loss and wouldstop using those paths for traffic

The Path MTU protocol packets are sent in transport reliable protocol(TRP) control packets. One consequence of this is that a loss of pathMTU probe packets in the network could cause a path to be consideredbad. This is not desirable because it is expected that the path MTUprobe packets could be lost, if the MTU is set above the actual MTU.This problem can be prevented by making a change to the TRP protocol.The TRP protocol detects loss by using sequence numbers and detectinggaps in received path sequence numbers. By specifying that the sequencenumbers (SNs) do not include the value of zero, SN=zero is allocated asan unused sequence numbers. The TRP protocol then use the number one asan initial sequence number and when the sequence number wraps around,the wrap around skips SN=0 and wraps around to 1. On the WAN egressprocessor in each APN node, a packet with a sequence number of SN=0 willnot be used in determining loss or re-sequencing. With these changes inplace, the path MTU probe packets can avoid accidentally making a pathbad by transmitting the packets with a path sequence number of 0. Eachpath has its own set of sequence numbers. A receiver waits for packetsto be in sequence before passing them up to the next process level. Forexample, if a receiver is waiting for a packet with a sequence number“10” but a packet with a sequence number of “11” comes in, the receiverwill hold on to the “11” packet until the “10” packet is received. Inoperation, the “10” packet is either retransmitted or considered lost,then the receiver passes the “11” packet to the next process level.Since path MTU probe packets use a sequence number of SN=zero, in effecthaving no sequence number, the receiver does not wait to order areceived path MTU probe packet. If the probe packet was lost or delayed,the receiver doesn't keep track of it. However, the path MTU probetrailer packets use the next path sequence number for that path.

FIG. 4D illustrates a probe trailer received out of order flow 470 inaccordance with the present invention. It is possible for a path MTUprobe and a path MTU probe trailer packets to be received out of orderas the WAN does not guarantee that all packets are received in the ordersent. Also, TRP packets are kept in order by the path sequence numberdescribed above. A problem may occur because path MTU probe packets havethe path sequence number set to zero, so that if a probe packet is lost,the path is not considered to be bad due to such loss. Generally, thesequence numbers allow packets lost on a path to be retransmitted. Thepath MTU packets except for the path MTU probe packet are retransmittedby the conduit if considered lost. This is advantageous because the pathMTU discovery process is able to operate with moderate packet losswithout hitting lots of timeouts and thus improves the speed of thediscovery process. Excessive packet loss would cause timeouts becausethe retransmitted packets could be lost as well as the retransmit of theretransmitted packets.

Regarding FIG. 4D, the APN node A 404 sends the path MTU probe packet410 followed by the path MTU probe trailer packet 414. When APN node B406 sees the path MTU probe trailer packet 472 first, the control planemodule 230 responds with sending a path MTU received probe trailer onlypacket 476 from APN node B 406. When APN node B 406 receives path MTUprobe packet 480 next, the control plane module 230 responds withsending a path MTU received probe packet 482 from APN node B 406. TheAPN node B 406 does not know whether the path MTU probe packet 480 was aretransmission as shown in FIG. 4C that made it through the network oran out of order situation. It does not matter in either case as thestate machine in the APN node B 406 handles each situation in the samemanner In this situation, the APN node A 404 initially receives a pathMTU receive probe trailer only packet 478 and transitions to a stateassociated with the path MTU probe was dropped by the network and thentriggers a retransmission of the path MTU probe, not shown in FIG. 4Dsince it will be ignored when received by the APN node B 406. The APNnode A 404 WAN receives the path MTU received probe 484 that is aresponse to the first path MTU probe packet 480. The control planemodule 210 would incorrectly assume that this is a response to thesecond path MTU probe packet. However, the APN node A 404 only needs toknow whether a packet of that MTU size could make it through the WANconduit 408 which was determined upon receipt of the path MTU receivedprobe 484. The reception of a subsequent path MTU receive probe due tothe second path MTU probe is not a problem and it is ignored since thecontrol plane module 210 will have moved on to probing a different MTU.

FIG. 5A illustrates an exemplary TRP control packet 500 formatted forMTU packets in accordance with the present invention. The TRP controlpacket 500 contains a Ethernet (ETH) header 502, an Internet Protocol(IP) header 503, a user datagram protocol (UDP) header 504, a transportreliable protocol (TRP) 505, a MTU probe packet 506, and padding to fitMTU size 507. The do not fragment bit is set to zero allowingfragmentation and the path sequence number is set to zero.

FIG. 5B illustrates an exemplary MTU probe packet 520 in accordance withthe present invention. The MTU probe packet 520 contains a control byte522 with a probe field in bit 7 set to a one indicating this is a probepacket and not a trailer, a request field in bit 6 set to a oneindicating this is a probe request, and a zero in bits 0-5 as a sequencenumber. The MTU probe packet 520 also contains a 16-bit send path index523, a 32-bit discovery instance 524, a 16-bit MTU 525, and padding 526filled up to MTU size. The MTU probe packet 520 corresponds to the pathMTU probe packet 410 shown in FIGS. 4A-4D and 412 in FIGS. 4A, 4C, and4D. The send path index is a unique number that is assigned to each pathin the APN. This number is distributed when the network is configured.This makes sure that the control plane can associate the packet to thecorrect path state machine. The discovery instance is incremented eachtime the probe not running state 602 is exited due to event 606 asdescribed with regard to FIG. 6 below. The control plane module on theWAN egress side of the path copies the discovery instance value into theresponses it sends to the WAN ingress side of the path. The controlplane module on the WAN ingress side of the path matches the receiveddiscovery instance to the current discovery instance and discards andignores packets that do not match. This makes sure that an olderretransmitted packet from the network does not interfere with the MTUprobing process.

FIG. 5C illustrates an exemplary MTU probe trailer packet 540 inaccordance with the present invention. The MTU probe trailer packet 540contains a control byte 542 with a probe field in bit 7 set to a zeroindicating this is a trailer, a request field in bit 6 set to a oneindicating this is a request, and a zero in bits 0-5 as a sequencenumber. The MTU trailer packet 540 also contains a 16-bit send pathindex 543, a 32-bit discovery instance 544, and a 16-bit MTU 545. Nopadding is used in the MTU probe trailer packet 540. The MTU probetrailer packet 540 corresponds to the path MTU probe trailer 414 and 420of FIGS. 4A and 4B, 414 in FIG. 4C, and 414 and 472 in FIG. 4D.

FIG. 5D illustrates an exemplary MTU received probe packet 560 inaccordance with the present invention. The MTU received probe packet 560contains a control byte 562 with a probe field in bit 7 set to a oneindicating this is a probe packet and not a trailer, a request field inbit 6 set to a zero indicating this is a probe reply, and a zero in bits0-5 as a sequence number. The MTU received probe packet 560 alsocontains a 16-bit send path index 563, a 32-bit discovery instance 564,a 16-bit MTU 565 selected from the path MTU probe, and a 16-bit IPdatagram length of actual probe packet received 566. The MTU receivedprobe packet 560 corresponds to the path MTU received probe packet 416and 418 of FIG. 4A and 482 and 484 of FIG. 4D.

FIG. 5E illustrates an exemplary MTU received trailer only packet 580 inaccordance with the present invention. The MTU received trailer onlypacket 580 contains a control byte 582 with a probe field in bit 7 setto a zero indicating this is a trailer, a request field in bit 6 set toa zero indicating this is a reply, and a zero in bits 0-5 as a sequencenumber. The MTU received trailer only packet 580 also contains a 16-bitsend path index 583, a 32-bit discovery instance 584, and a 16-bit MTU585 selected from the path MTU probe trailer. No padding is used in theMTU received trailer only packet 580. The MTU received trailer onlypacket 580 corresponds to the path MTU received probe trailer onlypacket 426 and 428 of FIG. 4B and 476 and 478 of FIG. 4D.

FIG. 6 is a path MTU discovery state machine 600 in accordance with thepresent invention. The path MTU discovery state machine 600 operates ona WAN ingress side of a path, such as WAN ingress processor modules 212and 234 at predetermined discover timeout intervals, such as every tenminute interval, to probe a conduit path and communicating nodes todetermine the highest MTU that can be used. Due to a system event orevents, a currently used MTU may not be optimum. The path MTU discoverystate machine 600 and MTU search process 700 described belowadvantageously select a smaller or a larger MTU to be used inoperational transmissions to improve system communication between nodesin the system.

The path MTU discovery state machine 600 is comprised of a probe notrunning state 602, a probing MTU state 604, a probing MTU sub-state A607, a probing MTU sub-state B 608, and a probing MTU sub-state C 609and transitions 606, 612, and 614 between states 602 and 604. Aninitialization event 605 places the path MTU discovery state machine ina probe not running state 602. Initialization events may include poweron of the APN node or a restart operation, such as may occur duringsoftware updating. Upon a discovery timeout, set for ten minutes forexample, an initial MTU value to probe is set and a transition 606 ismade to probing MTU state 604 where a first path MTU probe using theinitial MTU and first path MTU probe trailer are sent to a communicatingnode. If a path MTU received probe packet is received (got path MTUreceived probe) then a transition is made to probing MTU sub-state A607, an adjusted MTU is set with an increased MTU size, and the statemachine transitions back to state 604. A path MTU probe and path MTUprobe trailer are sent with the updated adjusted MTU size.

If a reply timeout has occurred or a path MTU received trailer onlypacket is received and in either case if a retry counter is less than 5,the state machine 600 transitions to probing MTU sub-state B 608. In thesub-state B 608 for the case that the reply timeout occurred, a replytimeout counter is incremented to note the timeout. Also, in thesub-state B 608, the adjusted MTU is set with the same MTU value as usedin the previous probe and the state machine transitions back to state604. A path MTU probe and path MTU probe trailer are sent with the sameMTU size as a previous MTU discovery instance. This aids indistinguishing random packet loss from packet discards due to an MTUthat is too large.

If a reply timeout has occurred or a path MTU received trailer onlypacket is received and in either case if the retry counter is greaterthan or equal to 5, the state machine 600 transitions to probing MTUsub-state C 609. In the sub-state C 609, since in either case the retrycounter is greater than or equal to 5, the MTU is determined to be toolarge and the adjusted MTU is set with a decreased MTU size and thestate machine transitions back to state 604. A path MTU probe and pathMTU probe trailer are sent with the updated adjusted MTU size.

In probing MTU state 604, a determination is made whether the search foroptimum MTU should end upon receiving indication that the path isdisabled taking transition 614 to the probe not running state 602. Also,in the probing MTU state 604 a determination is made whether there areno more MTU values to be searched. If no path MTU received probe packetswere received during the search, then the MTU of the path remainsunchanged when the system takes transition 612 to the probe not runningstate 602. In the case where an optimum MTU value has been determined tobe within the constraints of the process, as described in more detailwith regard to FIG. 7, the system takes transition 612 to the probe notrunning state 602. The next discovery timeout causes a transition 606 tobe taken back to check if any change has occurred that would affect thecurrent MTU value.

The WAN egress processing responds to a path MTU probe packet that isreceived or a path MTU probe trailer that is received. When a path MTUprobe packet is received for a path, the egress path data structure isupdated to reflect that a probe with a set MTU was received with acertain discovery instance value. The APN node then sends a path MTUreceived probe packet to the sender of the path MTU probe packet, overany available path. The IP length in the received path MTU probe is theactual size of the path MTU probe received. This is used in case theoriginal probe was fragmented.

When a path MTU probe trailer is received for a path, the egress pathdata structure is consulted to determine if a probe for that MTU withthat discovery instance has been received. If it has not, then a pathMTU received trailer only packet is sent in reply. If the egress pathdata structure indicates that a path MTU probe packet had been received,then nothing is done with the path MTU probe trailer packet.

FIG. 7 illustrates an exemplary MTU search process 700 in accordancewith the present invention. At step 702, a search for optimum MTUprocess is initiated with a selected initial probing MTU value, aselected initial maximum probing MTU value, and a selected initialminimum probing MTU value. Step 702 may include other processing insupport of the path processing such as having a discovery timeoutcounter that is used to initiate the path MTU discovery process. At step704, a path MTU probe and path MTU probe trailer are prepared with theinitial probing MTU value or an updated adjusted MTU value as describedwith regard to FIG. 6 and below in more detail. The probe packets arethen sent from an initiating node, such as APN node A 404 in FIG. 4A toa communicating node, such as APN node B 406 across a WAN conduit, suchas WAN conduit 408. The process 705 represents steps for finding thenext MTU size to probe or determining whether probing should be stopped.The path MTU received probe packet and the path MTU received probetrailer only packet are received in the WAN egress processor module 214and communicated to the control plane module 210 within the receivingAPN node A 404. At step 706, a determination is made whether a receivedprobe reply packet or message (msg.), such as the path MTU receivedprobe 418, was received in the initiating node. If a received probereply packet was received, the process 700 proceeds to step 708. At step708, a determination is made whether the received probe packet's IPdatagram length is less than the MTU sent at step 704. If the receivedprobe packet's IP datagram length is less than the sent MTU, the process700 proceeds to step 710. Such determination at step 708 indicatesfragmentation occurred and the IP datagram length may be used todetermine an MTU for future transmissions. From step 710, the process700 proceeds to step 702 to stop the current search for optimum MTU.Based on the predetermined interval, a new search for optimum MTU isinitiated. If the received probe packet's IP datagram length is equal tothe send MTU, the process 700 proceeds to step 712.

At step 712, an MTU probe success variable is set to the MTU of thesearch probe sent from step 704. At step 714, a determination is madewhether the MTU probe success value set in step 712 is equal to the maxprobing MTU. If the MTU probe success value is equal to the max probingMTU, the process 700 proceeds to step 710 and from there to step 702 tostop the current search for optimum MTU. Based on the predeterminedinterval, a new search for optimum MTU is initiated. If the MTU probesuccess value is not equal to the max probing MTU, the process 700proceeds to step 716. At step 716, the min probing MTU value is updatedto the current probing MTU value plus 1, to determine if a larger MTUvalue would be successful. At step 718, for packets without encryption,the current probing MTU value is set to a summed value of the maxprobing MTU value and the min probing MTU value divided by two, as anapproximation to a binary search for the best MTU to be used. Forpackets with encryption, the current probing MTU is adjusted downward tothe next advanced encryption standard block size. In one aspect, this isnecessary in the context of the range checking threshold T to make surethe best block size is selected. Alternative search methods may be usedin accordance with the present invention to advantageously increase ordecrease an MTU that is used in operational transmissions over time asevents occur in the communicating system.

At step 720, a determination is made whether the max probing MTU minusthe min probing MTU is less than or equal to a variable T. If the maxprobing MTU minus the min probing MTU is less than or equal to thevariable T, the process 700 proceeds to step 710 since the determinedMTU is within the tolerance of the system and is close enough to thebest MTU that could be determined. The variable T may be set to adifferent value as system improvements are made where such variations inMTU may be reduced. Advantageously, the use of the variable T is usefulin block encryption processes to account for the size of the encryptionblocks. Utilizing the T variable ensures that MTU sizes are not searchedfor that are within the same encryption block size. If the max probingMTU minus the min probing MTU is greater than T, then the process 700proceeds to step 722 which indicates a return to continue with thesearch for a best MTU to use in operational transmissions using theadjusted MTU at step 704.

Returning to step 706, if a received probe reply packet was notreceived, the process 700 proceeds to step 724. At step 724, adetermination is made whether a path MTU received probe trailer onlypacket was received. If a path MTU received probe trailer only packetwas not received, the probe packets may have been dropped. At step 725,the process 700 is informed to continue probes with the current MTU andreturns to step 704. If the path MTU receive probe trailer only packetwas received, the process 700 proceeds to step 726. At step 726, the maxprobing MTU is updated to a current probing MTU minus 1 to determine ifa smaller MTU value would be successful. At step 728, the currentprobing MTU value is set to a summed value of the max probing MTU valueand the min probing MTU value divided by two, as an approximation to abinary search for the best MTU to be used in operational transmissions.Alternative search methods may be used in accordance with the presentinvention to advantageously increase or decrease an MTU over time asevents occur in the communicating system. The process 700 then proceedsto step 720 as described above.

While the present invention has been disclosed in the context of variousaspects of presently preferred embodiments, it will be recognized thatthe invention may be suitably applied to other environments consistentwith the claims which follow.

We claim:
 1. A method in a network to address a probe trailer receivedout of order in a process to dynamically adjust a maximum transmissionunit (MTU), the method comprising: transmitting from a first networknode a first path MTU probe packet having a sequence number set to zeroand with padding to meet a packet length according to a selected MTU andallowing the first path MTU probe packet to be fragmented if the packetis too large; after said first transmitting step, transmitting from thefirst network node a first path MTU probe trailer only packet having apacket length significantly smaller than the selected MTU; at a secondnetwork node, receiving the first path MTU probe trailer only packet andtransmitting a path MTU received probe trailer only packet to the firstnetwork node; receiving, at the first network node and from the secondnetwork node the path MTU received probe trailer only packet andtransitioning to a state associated with the first path MTU probe packetwas dropped; at the second network node, after receiving the first pathMTU probe trailer only packet, receiving the first path MTU probepacket, in response to the sequence number being set to zero, refrainingfrom using the first path MTU probe packet in determining packet loss orretransmission, and transmitting a path MTU received probe trailer onlypacket to the first network node; at the first network node, afterreceiving the path MTU received probe trailer only packet, receivingfrom the second network node a path MTU received probe packet andtransitioning to the state associated with the first path MTU probepacket was received; and at the first network node, determining that anIP datagram size of the path MTU received probe packet is the same asthe selected MTU and less than a maximum probing MTU, and, in response,increasing the MTU and transmitting, from the first network node, asecond path MTU probe packet having an IP datagram size equal to theincreased MTU.
 2. The method of claim 1 further comprising: uponreceiving from the second network node the first path MTU probe traileronly packet, adjusting the MTU down from a maximum value; and uponreceiving from the second network node the path MTU received probepacket, adjusting the MTU back up to the maximum value.
 3. The method ofclaim 2 further comprising: utilizing the maximum value of the MTUduring operational communication between the first node to the secondnode.
 4. The method of claim 1 wherein a transport protocol allocateszero as an unused sequence number.
 5. The method of claim 1 wherein aretransmission of the first path MTU probe packet is triggered uponreceiving the path MTU received probe trailer only packet.
 6. The methodof claim 5 further comprising: ignoring reception of a subsequent pathMTU received probe packet.
 7. A computer readable non-transitory mediumstoring a computer program which causes a computer system to perform amethod in a network to address a probe trailer received out of order ina process to dynamically adjust a maximum transmission unit (MTU), themethod comprising: transmitting from a first network node a first pathMTU probe packet having a sequence number set to zero and with paddingto meet a packet length according to a selected MTU and allowing thefirst path MTU probe packet to be fragmented if the packet is too large;after said first transmitting step, transmitting from the first networknode a path MTU probe trailer only packet having a packet lengthsignificantly smaller than the selected MTU; at a second network node,receiving the path MTU probe trailer only packet and transmitting a pathMTU received probe trailer only packet to the first network node; at thesecond network node, after receiving the path MTU probe trailer onlypacket, receiving the first path MTU probe packet, in response to thesequence number being set to zero, refraining from using the first pathMTU probe packet in determining packet loss or retransmission, andtransmitting a path MTU received probe trailer only packet to the firstnetwork node; at the first network node, receiving from the secondnetwork node the path MTU received probe trailer only packet andtransitioning to a state associated with the first path MTU probe packetwas dropped; at the first network node, after receiving the path MTUreceived probe trailer only packet, receiving from the second node apath MTU received probe packet and transitioning to the state associatedwith the first path MTU probe packet was received; and at the firstnetwork node, determining that an IP datagram size of the path MTUreceived probe packet is the same as the selected MTU and less than amaximum probing MTU, and, in response, increasing the MTU andtransmitting, from the first network node, a second path MTU probepacket having an IP datagram size equal to the increased MTU.
 8. Acomputer readable non-transitory medium of claim 7 further comprising:upon receiving from the second network node the first path MTU probetrailer only packet, adjusting the MTU down from a maximum value; andupon receiving from the second network node the path MTU received probepacket, adjusting the MTU back up to the maximum value.
 9. The computerreadable non-transitory medium of claim 8 further comprising: utilizingthe maximum value of the MTU during operational communication betweenthe first node to the second node.
 10. The computer readablenon-transitory medium of claim 7 wherein a transport protocol allocateszero as an unused sequence number.
 11. A computer readablenon-transitory medium of claim 7 wherein a retransmission of the firstpath MTU probe packet is triggered upon receiving the path MTU receivedprobe trailer only packet.
 12. The computer readable non-transitorymedium of claim 7 further comprising: ignoring reception of a subsequentpath MTU received probe packet.
 13. A method in a network node todynamically adjust a maximum transmission unit (MTU), the methodcomprising: transmitting a path MTU probe packet with padding to meet apacket length according to a selected MTU and allowing the path MTUprobe packet to be fragmented if the packet is too large; subsequent totransmitting the path MTU probe packet, transmitting a path MTU probetrailer packet having a packet length significantly smaller than theselected MTU, the path MTU probe trailer packet having a currentmulti-bit discovery instance, wherein the current multi-bit discoveryinstance comprises a value that identifies a current instance of an MTUprobing process and that is incremented each time a new instance of theMTU probing process begins; receiving a path MTU received probe traileronly packet in response to a subsequent MTU discovery probe; comparing avalue of a multi-bit discovery instance field for said received path MTUreceived probe trailer only packet with the current multi-bit discoveryinstance field; in response to determining that the value of themulti-bit discovery instance field for said received oath MTU receivedprobe trailer only packet matches the current multi-bit discoveryinstance field, adjusting the MTU down; and in response to determiningthat the value of the multi-bit discovery instance field for saidreceived oath MTU received probe trailer only packet does not match thecurrent multi-bit discovery instance field, ignoring the received oathMTU received probe trailer only packet.
 14. The method of claim 13further comprising: discarding said received oath MTU received probetrailer only packet if the value of the multi-bit discovery instancefield does not match the current multi-bit discovery instance.
 15. Themethod of claim 13 further comprising incrementing the current multi-bitdiscovery instance each time a probe not running state is exited.